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Preface 


This  paper  provides  a  brief  summary  of  the  direct  costs  associated  with  automation.  It  is 
intended  to  provide  a  framework  for  designers,  managers,  and  pilots  in  implementing  measures 
to  mitigate  these  costs.  Safety  improvements  are  not  the  province  of  any  one  of  these  groups. 
Instead,  an  integrated  effort  between  these  communities  is  necessary  to  promote  aviation  safety. 
In  writing  this  paper  I  have  assumed  that  the  reader  has  a  working  knowledge  of  glass  cockpit 
aircraft,  as  well  as  a  basic  understanding  of  human  factors  issues.  In  order  to  narrow  the  scope  of 
this  project  I  have  focused  exclusively  on  automation  issues  arising  from  studies  of  transport 
aircraft.  In  spite  of  the  specific  focus  on  aviation,  the  issues  raised  here  apply  across  a  wide  range 
of  highly  automated  domains. 

I  would  like  to  thank  Lieutenant  Colonel  Steven  A.  Kimbrell  for  his  advice  and  assistance 
on  this  project. 
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Abstract 

Cockpit  automation  has  delivered  many  promised  benefits  such  as  improved  system  safety 
and  efficiency,  however,  at  the  same  time  it  has  imposed  system  costs  that  are  often  manifest  in 
the  forms  of  mode  confusion,  errors  of  omission,  and  automation  surprises.  An  understanding  of 
the  nature  of  these  costs  as  well  as  associated  influencing  factors  is  necessary  to  adequately 
design  the  future  automated  systems  that  will  be  required  for  Air  Mobility  Command  aircraft  to 
operate  in  the  future  air  traffic  environment.  This  paper  reviews  and  synthesizes  Human  Factors 
research  on  the  costs  of  cockpit  automation.  These  results  are  interpreted  by  modeling  the 
automated  cockpit  as  a  supervisory  control  system  in  which  the  pilot  works  with,  but  is  not 
replaced  by,  automated  systems.  From  this  viewpoint,  pilot  roles  in  the  automated  cockpit 
provide  new  opportunities  for  error  in  instructing,  monitoring,  and  intervening  in  automated 
systems  behavior.  These  opportunities  for  error  are  exacerbated  by  the  limited  machine 
coordination  capabilities,  limits  on  human  coordination  capabilities  and  properties  of  machine 
systems  that  place  new  attention  and  knowledge  demands  on  the  human  operator.  In  order  to 
mitigate  the  risks  posed  by  these  known  opportunities  for  error  and  associated  influencing 
factors,  a  system  of  defenses  in  depth  is  required  involving  integrated  innovations  in  design, 
procedures,  and  training.  The  issues  raised  in  this  paper  are  not  specific  to  transport  aircraft  or 
the  broader  aviation  domain,  but  apply  to  all  current  and  future  highly  automated  military 
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systems. 


Part  1 


Background 


The  laws  that  govern  the  behavior  of  human-machine  systems  are,  in  many  ways,  analogous 
to  the  laws  that  govern  our  physical  world.  Actions  that  affect  any  one  system  component 
invariably  have  ripple  effects  and  sometimes-unforeseen  interactions  with  other  system 
components.  For  example,  while  the  evolution  of  powerful  automated  cockpit  systems  has 
allowed  for  the  current  high  levels  of  safety  and  efficiency  in  the  aviation  system  it  has  also 
resulted  in  new  types  of  potentially  serious  system  failures  in  the  form  of  breakdowns  in  human- 
machine  coordination.  This  paper  discusses  observed  performance  problems  in  the  human- 
machine  cockpit  team  with  a  goal  of  providing  Air  Mobility  Command’s  (AMC’s)  designers, 
administrators,  and  operators  with  an  understanding  of  the  known  risks  associated  with  the 
automated  cockpit  systems  that  will  be  required  to  operate  in  the  future  air  traffic  environment. 
Based  on  this  assessment  of  known  risks,  an  integrated  set  of  measures  can  be  developed  to 
mitigate  the  risks  associated  with  the  introduction  of  these  automated  systems. 

The  evolution  of  highly  capable  automated  cockpit  systems  has  provided  substantial  benefits 
to  the  aviation  system.  Automated  cockpit  systems  are  the  driving  force  behind  the  safe,  precise, 
and  economical  operations  that  have  allowed  the  aviation  system  capacity  to  increase 
dramatically  over  the  last  50  years  while  providing  corresponding  increases  in  safety  and 
economy.  Studies  indicate  that  air  travel  is  one  of  the  safest  transportation  mediums  with  an 
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accident  rate  of  less  than  2  per  million  departures.'  Additionally,  the  introduction  of  automated 
navigation  systems  and  the  flight  management  computer  (FMC)  has  provided  substantial  fuel 
savings,  and  in  combination  with  automated  system  controllers  has  allowed  for  the  elimination  of 
the  navigator  and  flight  engineer,  thus  reducing  training  and  personnel  costs. 

Reports  estimate  that  over  the  next  ten  years  aviation  traffic  growth  will  continue  at  a  5% 
yearly  rate  .  Since  the  world  aviation  system  is  already  nearing  capacity,  significant  system 
changes  will  be  necessary  to  facilitate  this  anticipated  growth.  New  and  increasingly  powerful 
automated  systems  will  be  required  to  implement  the  future  air  traffic  environment.  For 
example,  “Free  Flight”  is  one  proposal  currently  under  development  to  increase  the  efficiency 
and  capacity  of  the  aviation  system  by  allowing  pilots  to  fly  random  routes  and  altitudes.  The 
implementation  of  such  a  system  will  require  the  addition  of  automated  cockpit  planning  and 
collision  avoidance  aids.  Although  many  AMC  aircraft  are  currently  undergoing  extensive 
cockpit  upgrades,  they  will  likely  require  further  upgrades  to  comply  with  the  requirements  of 
the  future  air  traffic  environment. 

In  spite  of  these  substantial  observed  and  potential  benefits,  cockpit  automation  also 
imposes  costs  on  the  aviation  system.  These  costs  are  frequently  expressed  in  the  form  of 
accidents  and  incidents  attributed  to  the  breakdowns  in  coordination  between  the  pilot  and 
automated  systems.  While  the  overall  rate  of  aviation  accidents  has  declined  dramatically  over 
the  last  30  years,  little  improvement  has  been  seen  over  the  last  15  years,  despite  the  continued 
evolution  and  improvement  of  automated  cockpit  systems  such  as  the  Ground  Proximity 
Warning  System  (GPWS)  and  Traffic  Collision  Avoidance  System  (TCAS)"'.  A  closer 
examination  of  the  aircraft  accident  data  indicates  that  human  error  accounts  for  between  65- 
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85%  of  all  accidents.  In  many  cases,  this  causal  human  error  can  be  attributed  to  inappropriate 
human  interaction  with  automated  systems.^ 

For  example,  on  24  April  1994  an  Airbus  300-600  crashed  while  on  approach  to  Nagoya, 
Japan.  During  the  approach,  the  copilot  inadvertently  engaged  the  aircraft’s  “Go  around  mode” 
which  caused  the  automated  systems  to  attempt  to  fly  away  from  the  ground  using  the  aircraft 
pitch  trim  system,  while  the  pilots  attempted  to  continue  the  landing  approach  via  input  to  the 
elevator.  The  pilots  were  unable  to  determine  that  the  pitch  trim  input  of  the  autopilot  system 
was  causing  difficulties  controlling  the  aircraft.  Additionally,  the  design  of  the  A300  autopilot 
(at  that  time)  did  not  allow  the  pilots  to  override  the  autopilot  by  use  of  opposing  control  stick 
pressure.  Thus,  the  pilots  and  automated  systems  continued  to  struggle  for  control,  with  the 
aircraft  eventually  pitching  up  to  near  vertical,  stalling  and  crashing  on  the  approach  end  of  the 
runway  killing  264  passengers  and  crew.^ 

This  accident  illustrates  a  phenomena  that  human  factors  researchers  term  “automation 
surprises”  -  i.e.  failures  of  the  human  operator  to  track,  monitor,  or  anticipate  the  actions  of 
automated  systems  leading  to  unintended  system  behavior.  A  better  understanding  of  the 
factors  that  contribute  to  these  automation  surprises  will  allow  AMC  to  determine  and  counteract 
the  risks  that  may  arise  from  implementation  of  new  automated  cockpit  systems.  This  paper  will 
discuss  the  human  role  in  automated  systems  and  review  the  factors  that  research  has  shown  may 
influence  breakdowns  in  coordination  between  human  and  machine  systems.  Based  on  these 
findings,  I  will  then  discuss  considerations  for  an  integrated  systems  approach  to  counteract  these 
risks.  Since  this  paper  focuses  on  automation  upgrades  to  AMC  aircraft,  I  will  focus  specifically 
on  cockpit  automation  in  transport  aircraft.  However,  these  findings  are  also  applicable  to  the 
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broader  aviation  domain,  as  well  as  other  highly  automated  systems  necessary  to  implement  the 
Armed  Services’  Joint  Vision  that  will  be  installed  in  a  wide  variety  of  military  systems. 


Notes 

*  Statistical  Summary  of  Commercial  Jet  Airplane  Accidents,  World  Wide  Operations  1959- 
i997.(Seattle,  WA:  Boeing  Commercial  Airplane  Group,  1998,.  13. 

^  Billings,  Charles  E.  Aviation  automation:  The  search  for  a  human  centered  approach. 
Mahwah,  NJ:  Lawrence  Erlbaum  Associates,  1997,  19. 

^  RTCA.  Report  of  the  RTCA  Board  of  Directors’  Select  Committee  on  Free  Flight. 
Washingon  DC:  Author.  1995. 

^  Statistical  Summary  of  Commercial  Jet  Airplane  Accidents,  World  Wide  Operations  1959- 
1997,  Seattle,  WA:  Boeing  Commercial  Airplane  Group,  1998,13. 

^  Billings,  Charles  E.  Aviation  automation:  The  search  for  a  human  centered  approach. 
Mahwah,  NJ:  Lawrence  Erlbaum  Associates,  1997,  4-5. 

^  Sekigawa,  E.,  and  M.  Mecham,.  “  Pilots,  A300  systems  cited  in  Nagoya  crash.”  Aviation 
Week  &  Space  Technology,  145  no.  5,  29  July,  1996,  36-37. 
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Part  2 


The  Human  Role  in  Automated  Systems 


In  order  to  understand  the  risks  of  automation,  we  must  first  understand  the  relationship 
between  humans  and  automated  systems.  While  it  is  tempting  to  assume  that  automated  systems 
function  independently  of  the  human  operator,  this  is  not  the  case.  As  Jordan  (1963)  first  noted 
over  35  years  ago,  humans  and  machines  are  not  independent  but  instead  are  complementary.^ 
They  must  work  together  to  achieve  desired  system  performance.  Even  the  most  highly 
automated  systems  still  require  the  presence  of  a  human  operator  to  monitor  system  performance 
and  intervene  in  the  case  of  system  abnormalities  and  emergencies.  In  order  for  humans  and 
machines  to  work  together  to  achieve  system  goals,  they  need  to  develop  or  engage  in  processes 
and  activities  that  ensure  coordination  and  avoid  conflict.  This  chapter  will  discuss  how  the  roles 
of  the  human  operator  in  highly  automated  systems  provide  new  opportunities  for  errors  and 
undesired  system  performance  associated  with  the  introduction  of  automated  systems. 

Prior  to  the  introduction  of  automated  cockpit  systems,  the  primary  role  of  the  pilot  was  to 
directly  control  aircraft  performance  by  continuous  inputs  via  the  flight  controls  and  throttles(s). 
As  automated  systems  have  become  more  powerful,  they  have  gradually  assumed  direct  control 
of  aircraft  performance,  while  the  pilot’s  role  has  shifted  to  a  monitor  of  automated  system 
performance.  For  example,  other  than  take  off  and  (usually)  landing,  most  of  the  direct  aircraft 
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control  in  a  modern  transport  aircraft  is  delegated  to  automated  systems  -  a  control  scheme 
known  as  supervisory  control. 

While  this  shift  in  roles  has  led  to  more  precise  and  economical  control  of  aircraft 
performance,  it  has  also  led  in  a  change  in  the  nature  of  observed  system  errors.  In  general,  since 
automated  systems  directly  control  aircraft  performance,  errors  of  commission  -  incorrect 
control  actions  have  decreased;  however,  errors  of  omission  -  failures  of  the  pilot  to  act  and 
intervene  when  required  have  increased."^  In  order  to  better  understand  the  reasons  behind  this 
trend,  and  also  in  order  to  predict  the  errors  that  may  be  observed  with  the  introduction  of  future 
automated  systems,  the  following  section  will  examine  in  detail  the  human  roles  in  supervisory 
control  systems  and  discuss  the  types  of  errors  that  may  arise  during  each  role. 


Figure  1  Human  roles  in  supervisory  control^ 

Figure  1  depicts  a  general  supervisory  control  system.^  In  this  type  of  a  system,  the  human 
operator  provides  higher  level  goals  to  the  automated  system  through  interaction  with  what  is 
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known  as  a  human  interactive  computer  (HIC).  The  HIC  provides  the  means  for  the  operator  to 
provide  control  instructions  and  monitor  system  behavior.  For  example,  on  the  modem  flight 
deck  pilots  provide  heading,  altitude,  and  routing  targets  through  both  the  flight  management 
computer  (FMC)  and  the  autopilot  Mode  Control  Panel  (MCP).  They  receive  feedback  on 
aircraft  performance  through  the  primary  flight  display  (PFD),  Navigation  Display  (ND)  and 
engine  indications  depicted  on  the  Engine  Indication  and  Crew  alerting  System  (EICAS).  (Eor 
those  unfamiliar  with  glass  cockpit  aircraft,  see  appendix  A  for  a  brief  description)  The  HICs,  in 
turn,  interpret  these  pilot  inputs  and  (based  on  environmental  conditions/aircraft  performance) 
provide  inputs  to  the  servos  that  actually  control  aircraft  performance  through  what  is  termed 
task  interactive  computers  (TICs).  Since  the  focus  of  this  paper  is  on  human-machine  interaction, 
I  will  focus  on  human  interactions  with  the  HIC. 

A  closer  examination  of  human  responsibilities  and  tasks  in  supervisory  control  systems 
reveals  potential  problems  for  the  operator’s  ability  to  coordinate  human  and  machine 
performance.  Sheridan  identifies  five  basic  human  roles  in  supervisory  control:  planning, 
teaching,  monitoring,  intervening,  and  learning. 

In  the  planning  role,  the  operator  decides  which  variables  to  manipulate,  develops  criteria  to 
assess  system  actions,  and  determines  constraints  on  activities.  The  planning  process  provides 
the  basis  for  instructing  automated  systems  and  monitoring  subsequent  system  behavior.  Eor 
example,  upon  receipt  of  an  Air  Traffic  Control  (ATC)  clearance,  the  crew  plans  by  determining 
which  autopilot  mode  and  EMC  or  MCP  input  will  be  required  to  execute  that  clearance.  Once  a 
plan  is  developed,  the  pilot  “teaches”  the  automated  systems  by  providing  the  appropriate 
targets/instructions  to  automated  systems.  After  providing  input  to  the  automated  systems,  the 
pilot  then  monitors  system  performance  to  ensure  the  system  is  performing  as  expected. 
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Monitoring  refers  to  all  activities  involved  in  adjusting  system  performance  in  response  to  small 
deviations  (trimming),  as  well  as  fault  detection  and  diagnosis.  In  the  current  cockpit,  the  pilot 
relies  primarily  on  information  presented  on  the  Primary  Flight  Display  (PFD)  and  Navigation 
Display  (ND)  to  monitor  system  performance.  These  instruments  give  indications  of  aircraft 
attitude,  altitude,  airspeed,  and  heading,  as  well  as  active  aircraft  mode(s)  and  command  targets. 
The  pilot  determines  whether/when  it  is  necessary  to  intervene  with  machine  performance  (due 
to,  for  example,  task  completion,  machine  requests  for  assistance,  or  undesired  system 
performance).  Finally,  based  on  the  given  plan,  inputs  to  the  system,  system  behavior,  and 
interventions  (if  any),  the  pilot  learns  lessons  that  may  be  applied  to  system  control  in  future 
situations. 

Errors  can  occur  at  each  of  these  five  steps.  During  the  planning  stage  errors  occur  when  the 
pilot  develops  an  inappropriate  plan  for  providing  data  to  the  automated  systems.  These  errors 
have  two  general  causes.  First  (as  is  also  the  case  in  non  automated  systems),  errors  can  occur 
when  pilots  fail  to  consider  all  available  information  (such  as  fuel  state  or  existing  weather) 
when  developing  the  plan.  Second,  they  may  occur  when  pilots  do  not  understand  how  the 
automated  systems  will  respond  to  the  plan.  These  failures  may  be  either  due  to  an  inadequate 
mental  model  of  system  operation,  or  due  to  a  failure  to  understand  the  current  operating  mode 
of  the  aircraft.  Research  indicates  that,  due  to  the  complexity  of  automated  systems,  pilots 
frequently  possess  a  faulty  knowledge  of  automated  system  action,  with  a  majority  of  pilots 
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surveyed  indicating  that  they  have  been  surprised  by  automated  system  actions.  Also,  pilots 
may  possess  the  relevant  knowledge,  but  may  be  unable  to  apply  that  knowledge  in  the  current 
context  -  a  phenomenon  known  as  inert  knowledge.^  For  example,  although  pilots  may  have 
been  trained  on  programming  holding  patterns  into  the  FMC  database,  they  may  be  unable  to 
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retrieve  and  apply  that  information  when  called  upon  to  do  so  during  flight.  Research  indicates 
that  inert  knowledge  is  a  frequent  problem  in  automated  systems,  especially  when  the  required 
actions  are  only  infrequently  performed.'*^  Finally,  pilots  may  develop  an  inappropriate  plan  due 
to  a  lack  of  knowledge  regarding  the  operating  mode  of  the  aircraft  -  a  phenomenon  known  as 
mode  error.  This  problem  is  particularly  critical  since  in  some  cases,  the  same  operator  input 
will  result  in  drastically  different  system  behavior  depending  on  the  operating  mode  of  the 
aircraft.  For  example,  an  A320  crashed  on  approach  to  Strasbourg  in  January  1992  when  the 
crew  attempted  to  program  the  aircraft  to  fly  a  3.0-degree  glide  path.  However,  due  to  the  active 
autopilot  mode,  the  input  was  interpreted  as  a  command  to  fly  a  3,300-foot  per  minute  descent 
rate.  As  a  result,  the  aircraft  crashed  several  miles  short  of  the  runway.  Mode  error  is  a  complex 
problem;  however,  it  is  often  tied  to  the  failure  of  cockpit  mode  indications  to  capture  pilot 
attention  as  well  as  the  occurrence  of  automatic,  or  uncommanded,  mode  transitions  dictated  by 
system  software. 

Errors  can  also  occur  in  the  teaching  role.  These  errors  generally  take  the  form  of  data  input 
errors  -  often  the  incorrect  entry  of  altitude/navigation  information.  This  type  of  error  is  a 
common  cause  of  deviations  from  ATC  instructions.  While  incorrect  data  entry  is  generally 
considered  easier  to  detect  than  the  development  of  an  incorrect  plan  ,  the  frequent  presence  of  a 
time  delay  between  data  entry  and  system  impact  can  act  against  error  detection.  For  example, 
navigation  data  entered  prior  to  taxi  may  not  affect  aircraft  performance  until  many  hours  into 
flight.  The  1983  Korean  AirLines  shoot  down  over  Russian  air  space  may  have  been  caused  by 
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such  a  data  entry  error. 

Errors  also  arise  during  pilot  monitoring.  These  errors  take  the  form  of  a  failure  to  detect 
deviations  from  desired  performance.  These  deviations  from  desired  performance  may  arise 
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from  inappropriate  plans,  incorrect  data  entry,  automated  system  malfunctions,  or  in  response  to 
changes  in  the  task  situation  (variations  in  temperature/wind,  system  failures,  etc.).  Basic 
psychological  research  indicates  that  humans  are  relatively  poor  monitors,  and  often  fail  to  detect 
critical  events. Flight  simulator  studies  confirm  that  once  tasks  have  been  delegated  to 
automated  systems,  human  monitoring  is  often  insufficient  to  detect  problems.'^ 

Beyond  human  monitoring  limitations,  there  are  two  additional  reasons  for  this  relatively 
poor  monitoring  performance.  First,  in  highly  automated  cockpits,  research  shows  that  pilots 
have  gone  away  from  a  general  instrument  scan  towards  an  expectations-based  monitoring 
strategy  in  which  they  check  specific  cockpit  indications  to  confirm  that  the  system  is  performing 
as  expected.  As  a  result,  pilots  are  less  likely  to  detect  automated  system  actions  that  go  beyond 
pilot  expectations.'^  For  example,  the  logic  in  some  FMC’s  dictates  that  when  a  change  is  made 
to  the  landing  runway,  all  current  vertical  constraints  are  deleted  since  they  may  no  longer  be 
appropriate.  Since  this  deletion  is  not  expected,  research  indicates  that  pilots  often  do  not  check 
for,  and  thus  do  not  detect,  this  situation.  Second,  automated  systems  often  provide  feedback 
that  is  not  sufficiently  salient  to  attract  pilot  attention.  One  feature  of  automated  systems  is  that 
they  present  more  information  than  the  pilot  can  process  in  the  time  available  (information 
overload).  In  the  absence  of  salient  indications  (i.e.  flashing  lights,  color  changes,  etc.)  pilots 
often  do  not  pay  attention  to  potentially  relevant  information.  For  example,  one  simulator  study 
found  that  nearly  25%  of  pilots  who  accessed  a  particular  FMC  page  containing  information 
required  to  detect  an  error  failed  to  detect  the  error  due  to  the  poor  layout  of  the  FMC  page  (a 
cluttered  display  full  of  numbers  of  similar  appearance).'^  In  a  related  manner,  the  design  of  the 
FMC  also  provides  barriers  to  detecting  undesired  performance.  The  FMC  contains  a  wealth  of 
performance  and  environmental  data,  but  can  only  show  a  very  small  portion  of  that  data  at  any 
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one  time.  This  feature  is  referred  to  as  the  keyhole  property,  and  places  additional  demands  on 

pilots  in  that  they  must  not  only  realize  that  they  need  a  particular  piece  of  information,  but  must 

20 

also  remember  where  that  piece  of  information  is  located  in  the  FMC  menu  structure. 

Even  if  errors  are  detected,  they  must  still  be  corrected  to  prevent  undesired  system 
performance.  In  order  to  successfully  intervene  in  undesired  system  behavior,  the  pilot  must 
correctly  assess  the  nature  of  the  problem  and  determine  an  appropriate  strategy  for  correcting 
the  problem.  It  must  be  realized  that  in  many  situations  (for  example  when  required  performance 
is  beyond  human  capabilities  -  e.g.  a  Category  III  ILS  approach),  the  pilot  cannot  simply  assume 
manual  control,  but  must  provide  additional  instructions  to  automated  systems  to  correct  the 
problem.  Intervention  errors  occur  when  the  pilot  cannot  figure  out  either  a)  why  a  problem  has 
occurred,  b)  what  to  do  to  correct  it,  or  c)  is  unable  to  take  corrective  action.  For  example,  in  the 
previously  cited  Nagoya  crash,  the  pilots  were  unable  to  determine  why  the  system  was 
exhibiting  the  observed  behavior  (inadvertent  selection  of  the  go-around  mode),  were  unable  to 
determine  the  correct  action  (disengage  the  autopilot),  and  were  prevented  by  system  design 
from  overriding  system  actions.  Research  indicates  that  pilots  often  fail  to  understand  system 
operation  and  are  often  do  not  understand  available  methods  of  correcting  undesired  system 
behavior.  In  addition  to  these  findings,  the  growing  inability  of  operators  to  override  system 
action  due  to  increased  machine  authority  is  particularly  troubling.  In  essence,  since  pilots  may 
lack  the  authority  to  override  system  actions  they  (or  the  designers)  must  be  able  to  understand 
before  hand  the  implications  of  selecting  a  given  course  of  action  -  something  that  may  be 
difficult  or  impossible,  particularly  if  aircraft  designers  have  failed  to  consider  a  given 
possibility.  For  example,  the  1988  crash  of  an  A-320  while  on  a  demonstration  flight  in  France 
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was  caused  by  the  inability  of  the  pilot  to  understand  and  override  preprogrammed  flight  limits 

22 

during  a  low  altitude,  low  speed  pass  unforeseen  by  system  designers. 

Finally,  errors  may  occur  in  the  learning  process  when  pilots  learn  the  wrong  lessons/fail  to 
learn  from  past  experiences  with  automated  systems.  This  will  result  in  the 
formation/perpetuation  of  an  inaccurate  mental  model  of  system  activity,  thus  creating 
difficulties  in  future  planning,  monitoring,  and  intervening  with  automated  systems. 


Table  1  Human  Roles  and  Opportunities  for  Error  in  Supervisory  Control 


Human  Role 

General  Dijficulty 

Caused  by 

Contributing  factors 

Planning 

Inappropriate  plan 
developed 

1. 

Failure  to  consider 
relevant  information 

a) 

Inadequate  mental 
model 

2. 

Failure  to  understand 

b) 

Inert  Knowledge 

automated  system 

c) 

Mode  errors 

Teaching 

Improper  data  entry 

Wrong  data/incorrect 
location 

a)  Time  delays 

Monitoring 

Failure  to  detect  the  need 

1. 

Human  monitoring 

a) 

Inadequate  mental 

to  intervene 

limits 

models 

2. 

Expectation  based 

b) 

Information 

3. 

monitoring 

Inadequate  feedback 

c) 

overload 

Lack  of  salient 
indications 

d) 

Keyhole  property  of 
FMC 

Intervening 

Mis  sed/incorrect 
intervention  in  undesired 

1. 

Inability  to 
understand:  Why 

a) 

Inadequate  mental 
models 

system  behavior 

2. 

the  problem 
occurred?  Or  what  to 
do  to  correct  it? 
Unable  to  correct 

b) 

Complex  systems 

Learning 

Failure  to  learn  from 

Inadequate  mental 

experiences 

model 
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Part  3 


Human-Machine  Coordination 

In  addition  to  the  general  opportunities  for  error  that  arise  from  the  supervisory  control 
process,  human  and  machine  coordination  abilities  and  requirements  also  create  automation 
induced  performance  costs.  The  following  section  will  define  and  describe  coordination,  and 
describe  inherent  human-machine  coordination  problems. 

Coordination  Theory  was  developed  at  the  MIT  Sloan  School  of  Management  to  describe 
coordination  and  cooperation  across  a  broad  range  of  activities  and  can  be  used  as  a  theoretical 
framework  for  understanding  human-machine  coordination.  According  to  this  theory, 
coordination  is  defined  as  the  “management  of  dependencies”  between  the  activities  and  goals  of 
actors.^  These  dependencies,  or  potential  conflicts,  can  take  many  forms  including  constraints  on 
shared  resources,  time  availability  and  scheduling,  restrictions  on  simultaneous  operations,  as 
well  as  incompatibilities  between  different  task  and  sub  task  elements.  In  simpler  terms,  human- 
machine  coordination  entails  the  processes  required  to  detect  and  resolve  conflicts  between  the 
goals  and  actions  of  pilots  and  automated  systems. 

The  variety  and  multiple  sources  of  goals  and  actions  in  a  typical  flight  complicate  cockpit 
coordination.  For  example,  pilot  goals  may  include  navigating  from  airport  A  to  airport  B, 
following  air  traffic  control  directives;  following  prescribed  procedures,  etc.  Automated  systems 
also  hold  a  wide  variety  of  goals.  The  pilots  provide  most  of  these  goals  such  as  heading. 
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airspeed  and  altitude  targets.  The  aircraft  designers,  however,  provide  some  goals.  For  example, 
autopilots  (and  even  aircraft  control  software  in  the  most  advanced  aircraft)  are  programmed  to 
fly  above  a  minimum  airspeed  and  below  a  maximum  airspeed  at  all  times.  In  addition  to  goals 
provided  by  the  operator  or  designer,  some  machine  goals  may  be  provided  by  other  human  or 
machine  agents.  For  example,  data  link  will  allow  for  direct  communication  between  cockpit 
automation  and  ground-based  human  and  machine  agents.  Since  pilots  may  be  less  aware  of 
the  goals  provided  by  designers  and  outside  agents,  coordination  of  these  goals  and  actions  may 
be  particularly  difficult. 

In  human-human  teams  coordination  is  a  cooperative  endeavor  in  which  all  parties  share 
information  on  ongoing  tasks  and  goals,  and  actively  seek  to  resolve  misunderstandings  or 
ambiguities.  Unfortunately,  automated  cockpit  systems  possess  limited  communication  and 
inferential  abilities  that  severely  constrain  true  cooperation  among  human  and  machine  agents."^ 
As  a  result,  automated  systems  are  unable  to  share  the  responsibility  for  coordinating  intentions 
and  actions  due  to  limited  machine  abilities  as  well  as  the  dynamic  nature  of  the  aviation  domain. 
Therefore,  pilots  are  primarily  responsible  for  detecting  and  resolving  present  and  future 
conflicts  between  human  and  machine  goals  and  actions.  This  implies  that  pilots  must 
understand  not  only  machine  goals  and  actions,  but  must  understand  how  automated  systems  will 
interpret  pilot  actions  as  well.  For  example,  in  the  Nagoya  crash,  human-machine  coordination 
broke  down  when  the  pilots  were  unable  to  ascertain  autopilot  goals  (to  climb  away  from  the 
ground)  and  actions  (autopilot  induced  nose-up  trim  inputs).  Additionally,  when  the  pilots 
recognized  that  something  was  wrong,  attempts  to  resolve  the  conflict  were  unsuccessful  because 
they  did  not  realize  that  since  the  autopilot  was  engaged  in  the  go-around  mode,  it  would  not 
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correctly  interpret  their  corrective  actions  (in  the  go-around  mode,  the  trim  system  locked  out 
pilot  nose  down  trim  inputs). 

Research  indicates  four  major  machine  communication  and  design  factors  that  contribute  to 
breakdowns  in  human-machine  coordination  -  an  inability  to  sense  operator  goals,  an  inability  to 
communicate  machine  goals,  an  inability  to  communicate  a  lack  of  clear  understanding  of 
operator  inputs,  and  an  inability  to  communicate  proximity  to  the  limits  of  automated  system 
capabilities.  First,  machines  generally  lack  an  ability  to  sense  operator  goals. ^  As  a  result, 
designers  are  forced  to  make  (sometimes-faulty)  assumptions  about  pilot  intentions  and  probable 
actions.  This  inability  to  sense  goals  has  been  shown  to  lead  to  a  variety  of  potential  breakdowns 
in  coordination.  For  example,  when  a  pilot  changes  the  designated  landing  runway  in  the  FMC,^ 
the  machine  does  not  know,  and  cannot  ask,  whether  or  not  this  runway  change  will  also  require 
a  change  to  the  previously  constructed  vertical  profile.  As  a  result,  the  system  design  is  forced  to 
make  an  assumption  about  pilot  intent.  Since,  in  many  cases,  a  change  in  runway  often  results  in 
a  landing  in  the  opposite  direction,  the  system  design  assumes  the  pilot  will  want  to  construct  a 
new  vertical  profile  and  thus  deletes  the  stored  vertical  profile.  However,  this  design  feature 
leads  to  problems  when  the  change  in  runway  is  merely  a  sidestep  to  a  parallel  runway.  In  this 
case,  ATC  expects  the  pilot  to  retain  the  vertical  constraints  automatically  deleted  by  the  FMC. 
If  the  pilot  does  not  realize  the  constraints  have  been  deleted  (as  research  shows  is  quite  often  the 
case)  the  result  is  a  failure  to  meet  an  assigned  altitude  restriction. 

Second,  automatic  systems  do  not  always  clearly  communicate  their  own  goals.  Automated 
systems  often  provide  a  great  deal  of  feedback  on  what  actions  they  are  taking,  but  little 
information  on  why  they  are  taking  those  actions.  Feedback  on  machine  goals  is  especially 
important  because  machine  actions  can  result  not  only  from  goals  provided  by  the  pilot,  but  also 
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from  goals  provided  by  system  designers  and  other  agents.  As  demonstrated  by  the  Nagoya 
crash,  pilots  may  be  unable  to  resolve  conflicts  without  knowledge  of  the  machine  goals  that  led 
to  the  observed  discrepant  behavior. 

Third,  unlike  human  crewmembers,  automated  systems  often  lack  the  ability  to  clarify 
ambiguous  or  misunderstood  instructions.  This  is  especially  true  when  full  understanding 

o 

requires  knowledge  of  other  pilot  goals  and  intentions.  For  example,  in  the  1995  crash  of  a 
B757  enroute  to  Cali,  Colombia,  the  crew  was  cleared  to  proceed  direct  to  a  point  named  ROZO. 
Due  to  crew  confusion  over  waypoint  designators,  the  crew  entered  “R”  into  the  FMC  instead  of 
the  required  “ROZO”.  Unfortunately  the  point  “R”  corresponded  to  a  NDB  located  near  Bogota, 
nearly  100  miles  in  the  opposite  direction  of  intended  landing.  Since  the  FMC  was  unable  to 
detect  the  ambiguity  inherent  in  a  command  to  turn  in  the  opposite  direction  of  the  landing 
airport,  the  FMC  dutifully  followed  this  command  and  executed  a  nearly  180  degree  course 
reversal  while  descending  in  mountainous  terrain  resulting  in  a  fatal  impact  with  terrain.^ 

Finally,  automated  systems  do  not  give  clear  indications  when  they  are  approaching  the 
limits  of  their  capability. While  human  performance  often  degrades  gradually,  thus  giving 
other  team  members  time  to  detect  and  compensate  for  impending  failure,  machine  systems  often 
give  up  suddenly  without  warning.  As  a  result,  pilots  may  have  insufficient  time  to  plan  and 
compensate  for  machine  failure.  For  example,  a  China  Airlines  B747  experienced  engine  failure 
off  the  coast  of  San  Francisco  engine  failure  above  3-engine  altitude  cruise  altitude.  While  the 
aircraft  slowly  decelerated,  the  autopilot  was  forced  to  provide  increasing  control  force  to  keep 
the  wings  level.  Since  the  indications  of  autopilot  effort  were  difficult  to  determine,  when  the 
crew  disengaged  the  autopilot  they  were  caught  off  guard  by  the  control  inputs  required  to  hold 
level  flight,  resulting  in  a  30,000-foot  altitude  loss  and  structural  damage  prior  to  recovery. 
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In  summary,  due  to  limited  machine  inferential  and  communication  abilities,  machines 
cannot  share  the  responsibility  for  coordinating  human  and  machine  actions.  As  a  result,  the 
need  to  coordinate  the  actions  of  automated  systems  place  additional  knowledge  and  attentional 
demands  on  the  human  operator.  In  other  words,  the  human  operator  is  responsible  knowing 
how  a  machine  will  act  in  a  given  situation,  but  must  be  able  to  monitor  for  pending  conflicts 
with/between  the  large  number  of  automated  cockpit  systems.  Many  of  these  conflicts  arise  due 
to  inherently  limited  machine  communication  and  inferential  abilities  that  cause  automated 
systems  to  misinterpret  or  make  incorrect  inferences  regarding  human  intentions. 
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Part  4 


Factors  Affecting  Human  Coordination  Activities 


Effective  integration  of  automated  systems  requires  an  understanding  of  the  factors  that  may 
serve  to  limit  the  pilot’s  capacity  to  meet  the  demands  of  coordinating  human  and  machine 
actions.  In  general,  research  indicates  that  most  breakdowns  in  human-machine  coordination 
occur  in  high  workload,  high  time  pressure,  unfamiliar  situations.^  The  following  sections  will 
discuss  the  role  of  time  pressure,  workload,  and  situation  awareness  in  leading  to  breakdowns  in 
human-machine  coordination. 


Workload 

The  amount  of  cognitive  workload  imposed  by  automated  systems  affects  the  pilot’s  ability 
to  program,  monitor,  and  intervene  with  automated  systems.  The  relationship  between  human 
performance  and  workload  generally  follows  an  inverted  U  shaped  function  known  as  the 
Yerkes-Dodson  law.  In  general,  human  performance  is  poor  in  conditions  of  both  low  and  high 
workload,  with  optimum  performance  occurring  at  moderate  levels.  Thus,  workload  can  harm 
human  performance  in  two  ways  -  by  either  raising  or  lowering  pilot  workload  away  from 
optimum  levels.  When  workload  is  too  low,  boredom  decreases  the  pilot’s  ability  to  monitor 
automated  systems.  Conversely,  when  workload  is  too  high,  a  phenomenon  known  as  cognitive 
tunneling  is  likely  to  occur  which  serves  to  limit  human  performance.  Under  cognitive 
tunneling,  humans  tend  to  focus  on  a  relatively  small  number  of  salient  cues  and  ignore  other 
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information  sources.  As  a  result,  pilots  may  fail  to  detect  the  need  to  intervene  in  automated 
system  performance,  or  may  be  unable  to  consider  the  factors  required  to  adequately  program 
automated  systems.  For  example,  cognitive  tunneling  is  one  explanation  for  the  1972  crash  of  an 
Eastern  Airlines  L-1011  in  the  Florida  everglades  in  which  preoccupation  with  a  burned  out 
landing  gear  indicator  prevented  the  crew  from  detecting  a  gradual  descent  into  the  terrain."^ 

While  automated  systems  are  often  intended  to  reduce  pilot  workload,  research  indicates  that 
the  introduction  of  glass  cockpit  aircraft  has  had  little  effect  on  overall  pilot  workload.  The 
introduction  of  automated  systems  tends  to  redistribute,  rather  than  reduce,  pilot  workload.^  The 
general  trend  for  cockpit  automation  is  to  reduce  workload  when  it  is  already  low  -  at  cruise  - 
and  increase  workload  when  it  is  already  high  -  during  departure  and  arrival,  a  phenomena 
known  as  “clumsy  automation”.^  This  workload  distribution  is  due,  in  large  part,  to  the  high 
cognitive  demands  imposed  by  planning  and  instructing  automated  systems,  i.e.,  the  demands  of 
data  entry  associated  with  the  frequent  route  and  altitude  changes  occurring  in  the  terminal  area. 

Time  Pressure 

The  previously  described  crash  of  a  B757  enroute  to  Cali,  Colombia  illustrates  the  effects  of 
time  pressure  on  the  pilot’s  ability  to  instruct  and  monitor  automated  systems.  In  this  case,  the 
crew  was  under  considerable  time  pressure  due  to  pilot  acceptance  of  an  unanticipated  clearance 
to  fly  a  straight  in  approach  to  the  south  (as  opposed  to  overflying  the  field  for  an  approach  to  the 
north).  When  the  automated  systems  were  mistakenly  programmed  to  fly  direct  to  the  Romeo 
NDB  (as  opposed  to  ROZO),  it  took  the  crew  almost  one  minute  to  realize  that  they  were 

•y 

proceeding  almost  180  degrees  off  of  the  desired  course. 

Time  pressure  has  several  effects  on  the  pilot’s  ability  to  interact  with  automated  systems. 
A  review  of  judgments  and  decision  making  under  time  pressure  found  that  as  time  pressure 


21 


increases:  1)  people  tend  to  use  less  information,  or  use  available  information  in  a  more  shallow 
manner,  2)  more  important  sources  of  information  are  given  increasing  weight,  3)  people  tend  to 
lock  in  on  one  strategy,  and,  as  a  result,  4)  performance  decreases.^  In  general,  these  studies 
suggest  that  as  time  pressure  increases,  pilots  will  consider  less  of  the  available  evidence  when 
instructing,  monitoring,  and  intervening  with  automated  systems  and  may  also  seek  less 
cognitively  demanding  methods  of  arriving  at  these  decisions  thus  leading  to  breakdowns  in 
human-machine  coordination.  For  example,  a  recent  simulator  study  found  that  time  pressure 
significantly  reduced  pilots’  ability  to  detect  problems  with  the  automated  implementation  of 
data  link  ATC  clearances.^ 


Situation  Awareness 

In  order  to  anticipate  the  actions  of  automated  systems,  the  pilot  must  have  good  situation 
awareness  -  i.e.,  knowledge  of  the  current  and  projected  aircraft  state  and  associated  variables. 
Since  automated  systems,  and  not  the  pilot,  actually  control  the  aircraft  the  pilot  may  fail  to 
develop  an  accurate  mental  picture  of  aircraft  state  and  important  information  needed  to  control 
the  aircraft.  Thus,  the  introduction  of  automated  systems  can  lead  to  poor  situation  awareness 
resulting  in  problems  instructing,  monitoring,  and  intervening  in  automated  systems.  Each  of 
these  will  be  discussed  in  turn. 

First,  poor  situation  awareness  can  lead  to  problems  instructing  automated  systems.  For 
example,  problems  of  mode  error  (i.e.,  pilot  actions  inappropriate  for  the  given  aircraft  mode 
such  as  the  previously  described  Strasbourg  accident)  can  attribute  to  poor  situation  awareness. 
Second,  as  indicated  previously,  pilot  monitoring  in  automated  aircraft  is  based  primarily  on 
expectations  of  aircraft  performance.^^  A  lack  of  situation  awareness  regarding  aircraft  state  or 
the  presence  of  potential  threats  can  lead  to  the  failure  to  detect  the  need  to  intervene.  For 


22 


example,  problems  with  situation  awareness  contributed  to  the  previously  described  Cali  crash. 
Due,  in  part,  to  the  automated  removal  of  certain  navigation  information  shown  on  the  cockpit 
displays  following  a  change  in  waypoints,  as  well  as  a  reliance  on  the  autopilot/FMC  for 

navigation,  the  crew  was  unsure  of  aircraft  position,  as  well  as  the  position  of  nearby  waypoints. 

12 

As  a  result,  the  crew  was  unaware  of  the  close  proximity  of  steeply  rising  terrain. 

Finally,  poor  situation  awareness  can  also  lead  to  problems  with  successfully  intervening  in 
automated  system  actions  -  a  phenomenon  known  as  “out  of  the  loop  syndrome.”  For  example, 
a  recent  simulator  study  found  that  the  introduction  of  an  automated  system  that  automatically 
loaded  data  link  clearances  into  the  FMC  and  MCP  resulted  in  a  decreased  pilot  knowledge  of 
current  aircraft  state,  thus  delaying  actions  to  intervene  in  undesired  performance.  In  this  study, 
pilots  using  this  automated  system  were  less  likely  to  detect  a  clearance  to  descend  to  an  altitude 
above  the  current  altitude.  When  the  problem  (an  unexpected  climb)  was  encountered,  the  lack 
of  knowledge  of  current  altitude  resulted  in  delayed  or  misdirected  intervention  (e.g.,  attempts  to 
trouble  shoot  a  presumed  faulty  autopilot).'"^ 

In  summary,  since  automated  systems  assume  the  role  of  directly  controlling  aircraft 
performance,  pilots  may  encounter  problems  developing  the  situation  awareness  required  to 
instruct,  monitor,  and  intervene  in  system  performance.  However,  the  introduction  of  automated 
systems  does  not  always  lead  to  decreased  situation  awareness.  Instead,  it  appears  that  the 
affects  on  situation  awareness  depend  on  the  interaction  with  changes  in  pilot  workload.  Studies 
involving  failure  detection  during  autopilot  coupled  instrument  approaches  shows  that 
automation  may  improve  pilot  performance  to  the  extent  that  it  decreases  workload  while  not 
detracting  from  the  system  feedback  available  to  the  pilot.  Other  research  suggests  that  the 
introduction  of  automated  systems  may  allow  pilots  to  develop  a  better  awareness  of  the  strategic 
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situation  (knowledge  of  the  general  route  of  flight  in  relation  to  other  waypoints/hazards)  since 
the  automated  systems  free  the  pilot  from  concentrating  on  the  tactical  details  of  controlling 
system  operation.'^ 

The  bottom  line  is  that  the  introduction  of  automated  systems  may  either  help  or  hurt  pilot 
performance,  depending  on  the  tradeoff  between  reductions  in  pilot  workload  and  potential 
decreases  in  pilot  awareness  of  system  operations.  Additionally,  the  introduction  of  automated 
control  systems  may  allow  for  reduced  pilot  awareness  of  the  details  of  systems  operations,  but 
the  reduction  in  workload  associated  with  automated  system  control  may  free  the  pilot  to  focus 
attention  on  higher  level  problems.  In  order  to  assess  the  positive  or  negative  effects  of 
automation  on  human  performance,  system  designers  and  operators  must  consider  the  effects  of 
automation  on  pilot  workload,  as  well  as  the  consequences  for,  and  relative  importance  of, 
tactical  and  strategic  situation  awareness. 
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Part  5 


Machine  Factors  Affecting  Human-Machine  Coordination 


In  addition  to  the  factors  which  have  a  direct  effect  on  the  pilot  performance,  research  also 
indicates  that  many  features  of  modern  automated  systems  also  contribute  to  breakdowns  in 
human-machine  coordination.  Automated  systems  have  evolved  from  simple  systems  that 
carried  out  relatively  uncomplicated  functions  (for  example  early  autopilot  systems  did  little 
more  that  hold  heading  and  altitude)  into  highly  powerful  agent-like  systems  that  carry  out 
multiple  functions  and  pursue  complicated  goal-oriented  tasks  (for  example  the  modern 
autopilot/  FMC  can  plan  and  execute  complicated  flight  path  trajectories).  Research  indicates 
that  these  highly  capable  modem  systems  possess  several  attributes  -  authority,  autonomy, 
complexity,  coupling,  and  low  observability  -  that  contribute  to  breakdowns  in  human  machine 
coordination.^  A  better  understanding  of  the  impact  of  these  factors  will  allow  for  designers  and 
operators  to  make  better  informed  design  decisions  regarding  those  factors  that  can  be  controlled, 
and  to  implement  more  effective  measures  to  control  the  risks  posed  by  those  factors  that  cannot 
be  designed  out  of  future  automated  systems. 

Authority 

2 

Authority  describes  the  ability  of  the  automated  system  to  override/block  human  input. 
Automated  system  authority  is  often  intended  to  prevent  unsafe  operation  (for  example,  systems 
that  prevent  over/underspeed  conditions)  or  is  intended  to  prevent  human  actions  from 
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interfering  with  automated  systems  operation  (for  example,  trim  systems  that  lock  out  pilot  input 
while  the  autopilot  is  engaged).  However,  high  levels  of  system  authority  may  also  prevent  the 
pilot  from  intervening  in  the  case  of  undesired  system  operation.  For  example,  in  the  previously 
described  A320  crash  during  a  flight  demonstration,  the  pilot  could  not  override  the 
preprogrammed  flight  limits  when  such  a  response  was  required  to  prevent  impact  with  the 
ground.^ 

At  a  general  level,  high  levels  of  machine  authority  may  place  the  pilot  in  what  is  known  as 
the  responsibility- authority  double  bind.  The  responsibility-authority  double  bind  occurs  when 
the  human  operator  has  the  responsibility  for  system  operation,  but  does  not  have  the  authority  to 
take  all  necessary  control  actions."^  Research  across  a  range  of  domains,  shows  that  this  split 
between  authority  and  responsibility  leads  to  poor  system  operation.^  In  the  case  of  the  modem 
cockpit,  the  pilot  in  command  has  the  legal  and  moral  responsibility  for  ensuring  safe  and 
effective  operations,  yet  in  some  cases,  may  lack  the  ability  to  override  the  actions  of  automated 
systems.  This  lack  of  authority  means  that,  in  order  to  coordinate  human  and  machine  actions, 
the  pilot  must  anticipate  some  conflicts  before  they  occur  since  he  or  she  may  not  be  able  to 
intervene  after  the  fact.  Given  the  complexity  of  automated  systems,  human  cognitive  limits,  the 
dynamic  nature  of  the  environment,  and  the  possibility  of  machine  malfunction,  it  will  be 
impossible  for  the  pilot  to  anticipate  machine  actions  in  all  situations.  While  good  system  design 
can  minimize  the  number  of  situations  in  which  pilots  may  have  a  legitimate  reason  to  override 
machine  actions,  analysis  indicates  that  designers  cannot  anticipate  every  possible  situation.^ 

There  are  three  general  ways  in  which  machines  may  limit  pilot  authority.  First,  the  most 
obvious  situation  occurs  when  pilots  are  physically  unable  to  override  machine  actions.  For 
example,  most  fly  by  wire  aircraft  incorporate  features  that  prevent  overspeeding/overstressing 
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the  airframe.  Second,  pilot  authority  is  also  limited  when  the  human  effort  required  to  override 
machine  systems  exceeds  his  or  her  capabilities.  For  example,  automated  decision  making  aids 
may  usurp  pilot  authority  when  the  complexity  of  their  decision  processes  exceeds  the  capacity 
of  the  human  operator  to  accurately  assess  the  validity  of  the  decision.  A  recent  study  of  pilot 
interaction  with  a  complex  cockpit  flight  planning  aid  showed  that  pilots  often  followed  risky 
flight  planning  suggestions  when  the  automated  system  failed  to  consider  the  projected  track  of 

o 

hazardous  weather.  Finally,  relative  difficulties  reprogramming  automated  systems  can  also 
limit  pilot  authority.  A  recent  study  of  glass  cockpit  pilots  found  that  over  75%  of  reported 
problems  overriding  automated  systems  dealt  with  difficulties  reprogramming  automated 
systems  rather  than  difficulties  assuming  manual  control.^ 

Autonomy 

Autonomy  refers  to  the  capability  of  automated  systems  to  operate  for  long  periods  of  time 
with  minimal  operator  input.  For  example,  once  programmed  during  pre-taxi  operations,  the 
FMC  can  provide  navigational  guidance  for  the  duration  of  the  flight.  System  autonomy  creates 
problems  by  increasing  the  time  delay  between  control  input  and  associated  systems  response. 
As  this  time  delay  increases,  the  probability  of  error  detection  decreases.''  In  essence,  since 
human  memory  decays  with  time,  it  becomes  increasingly  difficult  for  the  operator  to  generate 
the  expectations  required  to  effectively  monitor  system  performance.  This  problem  is 
exacerbated  in  long  haul  flights  in  which  relief  crewmembers  swap  out  during  flight.  In  this 
case,  the  relief  crewmembers  monitor  system  behavior  that  may  be  the  result  of  inputs  made  by  a 
different  crewmember.  That  input  may  perhaps  be  incorrect  or  may  use  techniques  not  desired 
by  the  present  crew. 
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Complexity 


As  automated  systems  grow  more  powerful,  they  are  also  more  complex  both  in  terms  in  the 
number  of  automated  components  as  well  as  the  calculations  required  to  produce  system 
behavior.  This  complexity  makes  it  difficult  or  impossible  for  the  pilot  to  understand  and 
predict  system  behavior.  Since  an  appropriate  mental  model  of  automated  system  operations  is 
required  for  instructing,  monitoring,  and  intervening  in  system  behavior,  system  complexity  can 
lead  to  problems  in  each  of  these  areas.  Recent  surveys  indicate  that  pilots  often  do  not 
completely  understand  the  operation  of  automated  systems,  often  leading  to  instances  of 
undesired  system  behavior.  The  effects  of  complexity  on  the  pilot’s  ability  to  control  system 
behavior  may  interact  with  pilot  experience.  A  recent  study  found  that  pilots  with  600-  1500  hrs 
in  the  current  airframe  were  least  likely  to  detect  the  inappropriate  behavior  of  an  automated  data 
link  system.  This  may  be  due  to  a  relative  inability  to  generate  an  adequate  set  of  expectations 
regarding  system  behavior  as  a  result  of  a  limited  basis  of  personal  experience  which  is  not 
sufficient  to  compensate  for  lessons  forgotten  since  going  through  initial  training.'"^ 

Coupling 

Closely  related  to  complexity,  coupling  refers  to  interconnections  between  system 
components. Many  automated  systems  components  receive  inputs  from  and  give  commands  to 
a  number  of  interrelated  subsystems.  For  example,  figure  2  indicates  the  relation  of  the  FMC  to 
other  cockpit  systems.  Coupling  contributes  to  breakdowns  in  human-machine  systems  by 
limiting  the  pilot’s  ability  to  generate  an  accurate  mental  model  of  automated  system  actions. 
This  interferes  with  instructing,  monitoring,  and  intervening  with  automated  systems.  This  is 
especially  true  since  coupling  often  leads  to  automated  systems  doing  more  than  expected  by 
pilots  -  a  situation  that  is  particularly  difficult  to  detect.  Research  shows  that  since  pilots  monitor 
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systems  primarily  on  the  basis  of  their  expectations  of  system  behavior,  pilots  often  do  not 
monitor  for  and  thus  do  not  detect  these  situations.'^ 


Figure  2  Systems  Coupled  to  the  FMC 


Low  Observability 

Automated  systems  often  provide  inadequate  feedback  regarding  their  actions.  The  problem 
is  not  that  indications  do  not  exist,  rather  the  problem  is  that  the  indications  that  do  exist  require 
an  excessive  amount  of  effort  for  the  pilot  to  monitor  and  process  -  a  phenomenon  known  as  low 
observability.  It  is  not  enough  to  merely  present  information,  instead,  given  the  large  amount 
of  information  available  to  the  operator;  the  system  must  draw  operator  attention  to  the 
information  and  present  the  information  in  a  manner  that  requires  little  effort  to  understand.  For 
example,  in  the  Air  China  incident,  the  information  required  to  allow  pilots  to  determine  that  the 
autopilot  was  reaching  the  limits  of  its  control  authority  was  available  to  the  pilots,  but  it  was  not 
observable.  Since  the  cockpit  displays  did  not  draw  pilot  attention  to  the  relevant  information. 
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the  pilots  had  to  not  only  realize  the  need  to  check  the  data,  but  also  had  to  remember  where  the 
information  was  displayed.  Additionally,  the  pilots  would  have  had  to  take  the  additional  step  of 
comparing  actual  indications  to  their  memory  of  the  autopilot  limits,  since  the  display  did  not 
indicate  the  limits  of  autopilot  authority. 

In  addition  to  monitoring  difficulties,  low  observability  can  also  lead  to  problems  instructing 
and  intervening  in  automated  system  behavior.  In  the  instruction  phase,  mode  errors  often  occur 
when  pilots  are  not  aware  of  the  current  aircraft  mode  as  pointed  out  by  the  previously  described 
Strasbourg  crash.  Additionally,  as  pointed  out  by  the  Nagoya  Crash,  the  inability  to  determine 
the  current  mode  status  can  lead  to  an  inability  to  successfully  intervene  in  automated  system 
behavior. 

In  summary,  the  attributes  of  many  highly  capable  automated  systems  can  contribute  to 
problems  instructing,  monitoring  and  intervening  in  automated  system  action.  These  factors  all 
work  against  the  pilot’s  ability  to  develop  an  accurate  mental  model  of  system  behavior  and 
make  it  difficult  to  predict  future  behavior.  While  low  observability  can  be  corrected  through  the 
application  of  appropriate  human  factors  principles,  system  design  is  less  able  to  limit  the  effects 
of  coupling,  complexity,  autonomy,  and  authority  of  automated  systems.  Instead,  a  design 
decision  must  be  made  whether  the  risks  associated  with  a  give  system  can  be  sufficiently 
countered  by  a  systematic  attempt  to  control  the  risks  associated  with  implementing  new 
automated  systems. 
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Part  6 


Mitigating  the  Risk  of  Automated  Systems 


There  is  no  silver  bullet  in  the  effort  to  mitigate  the  risks  posed  by  automated  systems. 
Since  accidents  and  incidents  are  not  isolated  actions,  but  instead  the  product  of  a  chain  of 
events,  preventative  efforts  must  focus  on  a  variety  of  actions.  James  Reason’s  model  of  system 
failures  and  defenses  in  depth  provides  a  useful  means  of  visualizing  this  process.  Figure  3 
depicts  accidents  and  incidents  as  the  process  in  which  managerial  decisions  (budgeting, 
priorities,  hiring  practices)  serve  as  enabling  conditions  for  unsafe  acts  which  are  expressed  as 
accidents  when  they  pierce  weaknesses  and  failures  in  a  series  of  system  defenses  (design, 
training,  procedures,  etc.)  designed  to  guard  against  system  failures.^  In  essence,  accidents  and 
incidents  result  when  a  chain  of  actions  form  a  vector  that  penetrates  these  defenses. 
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Figure  3  Defenses  in  depth^ 

Since  a  complete  treatment  of  this  model  is  beyond  the  scope  of  this  paper,  I  will  focus  on 
the  defenses  in  depth,  rather  than  decisions  made  at  various  managerial  levels.  In  essence,  the 
defenses  in  depth  can  be  thought  of  as  features  of  design,  procedures,  and  training,  which  are 
aimed  to  mitigating  the  risks  identified  in  the  previous  chapters.  The  following  chapter  will 
discuss  general  considerations  in  system  design,  training,  and  procedures  in  order  to  mitigate  the 
risks  identified  in  human-machine  coordination,  limits  on  human  performance,  and  factors  of 
machine  design. 


Difficulties  in  Human-Machine  Coordination 

This  paper  identified  four  basic  problems  with  human  machine  coordination  activities, 
machines  cannot:  a)  sense  operator  goals,  b)  communicate  their  own  goals,  c)  identify/correct 
misunderstandings,  and  d)  communicate  when  approaching  the  limits  of  their  capability.  A 


34 


series  of  overlapping  design,  procedural  and  training  measures  must  be  employed  to  counteract 
these  coordination  problems. 

Current  research  in  cockpit  automation  addresses  design  solutions  to  these  problems.  In 
order  for  automated  systems  to  share  the  responsibility  for  coordinating  human  and  machine 
actions,  they  must  possess  a  better  knowledge  of  pilot  goals  and  actions.  The  “Agenda 
Manager”  is  one  effort  in  this  direction.  This  system  uses  information  about  pilot  statements  and 
actions  to  infer  pilot  goals.  It  then  compares  these  goals  to  the  goals  and  actions  of  automated 
systems  to  detect  conflicts  and  identify  potential  misunderstandings.  While  this  effort  is  only 
partially  successful  to  date,  it  represents  an  important  direction  for  future  design. 

Flight  procedures  must  also  compensate  for  the  inability  of  machine  systems  to  participate  in 
the  coordination  process.  Procedures  which  require  the  second  crewmember  to  confirm  inputs  to 
automated  systems  are  one  step  in  this  direction.  However,  research  indicates  that  since  both 
crewmembers  share  a  similar  awareness  of  the  environment  and  the  automated  system,  relatively 
few  errors  are  caught  by  the  second  crewmember."^  Therefore,  procedures  must  consider  the 
contribution  made  by  other  components  of  the  aviation  system  such  as  Air  Traffic  Controllers 
and  dispatchers. 

Finally,  training  must  emphasize  and  demonstrate  the  limits  of  machine  coordination  and 
communication  abilities.  In  order  to  train  pilots  on  these  machine  limits,  they  must  be  exposed  to 
these  situations  in  a  realistic  manner.  Unfortunately,  since  simulator  time  is  limited  (and 
expensive)  it  is  often  impossible  to  provide  this  training  in  the  simulator.  Often,  it  is  assumed 
that  pilots  will  pick  this  knowledge  up  during  line  operations.  As  an  alternative,  the  technology 
exists  to  replicate  important  cockpit  control  systems  on  an  interactive  desk-top  part-task  trainer. 
Using  this  technology,  pilots  could  learn  machine  coordination  limitations  via  a  set  of  predefined 


35 


illustrative  scenarios  flown  on  desk  top  computers  in  flying  units  or  at  home  on  personal 


computers. 


Human  limitations 

This  paper  identified  three  major  limitations  to  the  human’s  ability  to  control  automated 
systems  -  time  pressure,  workload,  and  situation  awareness.  From  a  design  standpoint,  care  must 
be  taken  to  look  not  only  at  the  effects  on  overall  workload,  but  also  on  workload  distribution. 
Workload  should  not  be  allowed  to  concentrate  in  areas  that  are  already  effort  intensive. 
Displays  must  also  be  designed  to  support  situation  awareness  by  indicating  elements  of  current 
and  future  aircraft  performance.  Research  in  mode  error  indicates  the  relative  importance  of 
highlighting  performance  and  mode  changes.^  Procedures  must  also  be  designed  to  compensate 
for  known  human  limitations.  For  example,  since  human  performance  is  substantially  degraded 
under  time  pressure,  procedures  should  be  designed  to  promote  adequate  time  available  to 
program/monitor/intervene  with  automated  systems.  Additionally,  since  monitoring  is  often 
ineffective  in  detecting  the  need  to  intervene  in  automated  system  actions,^  procedures  should 
focus  attention  on  fostering  error  detection  during  the  instruction  process.  Finally,  training  must 
stress  and  demonstrate  the  importance  of  time  and  workload  management. 

Machine  Factors 

The  autonomy,  authority,  complexity,  coupling,  and  low  observability  of  many  automated 
systems  make  it  difficult  for  the  pilot  to  develop  an  accurate  mental  model  of  automated  systems. 
From  a  design  point  of  view,  the  problem  with  automation  is  “...inappropriate  feedback  and 
interaction,  not  overautomation.”  Automated  systems  often  do  not  provide  adequate  feedback 
on  their  actions  or  intentions.  As  a  result,  the  pilot  cannot  develop  an  adequate  mental  model  of 
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system  operations  required  to  instruct,  monitor,  and  intervene  in  system  activities.  Inappropriate 
interaction  occurs  when  automated  systems  do  not  support  pilot  efforts  to  coordinate  or  override 
automated  actions.  Often  automated  systems  must  be  complex,  coupled,  and  autonomous  in 
order  to  accomplish  their  intended  roles.  As  a  result,  design  efforts  should  focus  on  system 
observability  (feedback)  and  coordinative  abilities  (authority). 

There  are  three  general  approaches  to  address  problems  with  system  observability,  all  of 
which  seek  to  minimize  the  effort  required  to  interpret  displayed  information.  First,  automated 
systems  must  communicate  both  what  they  are  doing  and  why  they  are  doing  it.  For  example, 
the  development  of  vertical  situation  displays  (VSD’s)  will  allow  pilots  to  better  understand  and 
visualize  vertical  path  information.^  Also,  given  the  autonomy  and  authority  of  automated 
systems  which  may  prevent  or  deter  pilot  intervention  after  the  fact,  feedback  must  be  provided 
on  the  future  intentions  and  actions  of  automated  systems,  especially  in  regards  to  mode 
changes.^ 

Second,  feedback  must  also  be  designed  to  reduce  the  effort  required  to  detect  relevant 
information  and  ascertain  its  importance.  Automated  systems  must  draw  pilot  information  to  the 
relevant  information  by  changes  in  color,  intensity,  auditory  alerts,  etc.  This  is  especially  critical 
when  access  to  the  relevant  information  is  likely  to  require  pilot- activated  display  changes  to 
view  the  information  -  for  example  information  on  the  EICAS  or  in  the  FMC.  Additionally, 
information  should  be  formatted  to  allow  easy  processing.  For  example,  when  integration  of 
several  parameters  is  required  (such  as  evaluating  a  given  vertical  path)  numeric  information  is 
often  more  difficult  to  process  than  a  graphic  depiction  of  the  same  data.^'^ 

Finally,  design  must  also  carefully  consider  any  limitations  on  pilot  authority.  As  discussed 
previously,  these  limits  may  be  either  due  to  the  physical  inability  to  override  automated 
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systems,  or  may  arise  when  pilot  override  is  possible,  but  limited  by  difficulties  in  understanding 
machine  actions  or  determining  the  desired  method  of  altering  machine  performance.  When 
pilots  cannot  adequately  assess  the  acceptability  of  machine  actions  they  will  likely  either  blindly 
accept  machine  actions,  or  inappropriately  intervene  in  correct  machine  behavior.  In  order  to 
address  these  problems,  design  (and  test/evaluation)  must  identify  and  minimize  limits  to  pilot 
authority.  When  pilot  authority  is  limited,  the  benefits  (i.e.  stall  prevention)  must  outweigh  the 
potential  costs.  Additionally,  once  limitations  to  pilot  authority  have  been  identified,  procedures 
and  training  should  be  developed  to  identify  and  preclude  intentional  operation  in  these  regions. 
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Part  7 


Conclusion 

While  the  introduction  of  highly  capable  automated  cockpit  has  provided  important 
contributions  to  system  safety,  precision,  and  efficiency,  it  has  also  imposed  system  costs  in  the 
forms  of  new  opportunities  for  error  which  are  expressed  as  breakdowns  in  human-machine 
coordination.  With  the  introduction  of  automated  systems,  the  role  of  the  pilot  has  changed  from 
system  controller  to  system  supervisor  responsible  for  instructing,  monitoring,  and  intervening 
with  automated  systems.  An  analysis  of  these  roles  as  well  as  the  capabilities,  limitations,  and 
characteristics  of  human  and  machine  members  of  the  cockpit  team  reveals  the  areas  of  greatest 
risk  to  system  safety  and  performance  that  may  be  associated  with  the  introduction  of  automated 
cockpit  systems  required  for  AMC  aircraft  to  operate  in  the  future  air  traffic  environment. 

In  general,  problems  arise  from  the  growing  power,  but  limited  coordination  and 
communication  abilities  of  current  automated  systems.  As  a  result,  the  human  operator  is  solely 
responsible  for  ensuring  cooperation  and  resolving  conflict  between  human  and  machine 
intentions  and  actions.  Time  pressure,  workload,  and  problems  with  situation  awareness  can 
reduce  the  pilot’s  ability  to  execute  these  coordination  responsibilities.  Additionally,  the 
autonomy,  authority,  complexity,  coupling,  and  low  observability  of  automated  systems  make  it 
difficult  for  the  pilot  to  understand  and  anticipate  machine  intentions  and  actions.  In  order  to 
compensate  for  these  known  risks  of  automated  systems,  AMC  must  implement  a  system  of 
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defenses  in  depth  utilizing  design,  training,  and  procedural  solutions  aimed  at  controlling  the 
previously  identified  risk  factors. 

The  issues  addressed  in  this  paper  are  not  unique  to  transport  aircraft  or  the  braoder  aviation 
domain,  but  instead  generalize  to  any  system  that  incorporates  highly  powerful,  agent-like 
automated  systems.  One  large  area  of  future  concern  is  the  Armed  Services’  Joint  Vision  which 
will  require  the  integration  of  information  technology  into  a  host  of  military  systems'.  This 
integration  will  depend  heavily  on  automation  to  integrate,  manage,  process,  and  synthesize  large 
volumes  of  information.  The  issues  identified  here  provide  an  important  first  step  for  ensuring 
that  we  realize  the  advantages  of  these  systems  while  mitigating  the  new  opportunities  for  error 
that  will  be  inherent  in  the  information  revolution. 
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Appendix  A 


A  Brief  Description  of  the  FMC  and  MCP 

The  Flight  Management  Computer  (FMC) 

The  FMC  Control  Display  Unit  (FMC  CDU)  is  the  pilot’s  interface  with  a  multifunction 
computer  system  (the  FMC)  that  allows  the  pilot  to  plan,  navigate,  and  control  the  aircraft. 
Through  interconnections  with  a  number  of  onboard  systems  and  sensors,  FMC  planning 
features  provide  the  pilot  with  weather  (winds/temperature),  fuel,  timing,  and  performance  data 
(optimal  altitudes,  takeoff  and  landing  speeds,  etc.).  The  FMC  also  contains  a  worldwide  data 
base  of  navigational  and  instrument  approach  data  that,  when  combined  with  satellite  or  inertial 
position  information,  allows  the  pilot  to  determine  aircraft  position  as  well  as  the  relative 
position  of  other  navigational  waypoints.  Finally,  interfaces  with  the  autopilot  and  automatic 
throttle  systems  allow  the  FMC  (depending  on  mode)  to  provide  steering,  altitude  and  speed 
commands  to  these  systems. 
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FMC  CDU 


Figure  4  A  typical  FMC  CDU 

The  FMC  CDU  allows  the  pilot  to  input  or  review  data  via  a  menu  driven  arehitecture. 
Figure  4  represents  the  FMC  CDU  similar  to  the  one  the  Boeing  B757  aireraft.  Data 
presentation  is  limited  to  approximately  12  lines  of  data  arranged  on  either  side  of  the  display 
unit.  In  order  to  support  the  wide  range  of  functions  available,  the  FMC  employs  a  branching 
menu  structure  in  which  pilots  can  access  by  selecting  the  appropriate  function  key  (Legs,  Route, 
Cruise,  etc.)  on  the  associated  data  entry  panel.  Once  a  given  function  is  selected,  the  pilot  can 
navigate  through  the  associated  menu  pages  by  using  the  “prev  page”  and  “next  page”  buttons. 
Although  there  are  several  different  manufacturers,  the  underlying  architecture,  controls,  and 
visual  presentation  are  highly  similar  across  different  FMC  CDU  units. 

The  FMC  CDU  allows  the  pilot  to  input  a  desired  route  of  flight,  vertical  profile,  and  speed 
profile.  Route  of  flight  information  may  be  entered  as  waypoints  (each  flight  is  composed  of  a 
set  of  many  waypoints)  on  the  appropriate  page  of  the  FMC  CDU  via  either  manual  keyboard 
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entry  or  selection  of  pre-stored  data  base  options  via  the  line  select  keys  adjacent  to  the  display 
screen.  Altitude  constraints  (either  cruise  altitude  or  a  restriction  to  cross  a  horizontal  waypoint 
or  altitude  at  a  given  airspeed)  may  also  be  entered  in  the  same  manner.  Aircraft  speed  may  be 
controlled  by  either  directly  entering  a  speed  value  on  the  appropriate  page,  or  by  selecting  a 
default  speed  profile  (based  on  fuel  economy  or  range  considerations). 

The  Mode  Control  Panel  (MCP) 

The  FMC  CDU  is  not  the  only  means  by  which  the  pilot  can  control  aircraft  speed,  heading, 
and  altitude.  The  MCP  (see  figure  5)  allows  the  pilot  to  control  autothrottle  and  autopilot  modes, 
as  well  as  to  provide  heading,  altitude,  air  speed,  and  vertical  speed  targets  to  these  systems. 
Autopilot  and  autothrottle  modes  are  selected  by  depressing  the  appropriate  buttons  (e.g.  LNAV 
(Lateral  Navigation),  VNAV  (Vertical  Navigation),  FLCH  (Flight  Level  Change),  etc.),  while 
airspeed,  altitude,  heading  and  vertical  speed  values  are  entered  into  the  appropriate  window  via 
the  associated  selector  knob.  Although  the  distinction  is  not  perfect,  the  FMC  CDU  is 
considered  a  “strategic”  interface  while  the  MCP  is  considered  a  “tactical”  interface.'  The  FMC 
CDU  is  often  used  to  implement  actions  that  will  take  place  or  continue  relatively  far  into  the 
future  (e.g.  entering  changes  to  the  route  of  flight),  while  the  MCP  is  often  used  to  implement 
more  immediate  actions  such  as  flying  an  assigned  heading  or  climbing  to  a  given  altitude.  Like 
the  FMC,  there  are  differences  among  manufacturers  and  models.  However,  at  a  conceptual  level 
most  MCP  functions  are  very  similar. 
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Mode  Control  Panel  (MCP) 

Figure  5  A  Typical  Mode  Control  Panel 

In  order  to  control  aircraft  performance  via  the  MCP,  the  desired  target(s)  must  be  entered 
into  the  appropriate  window(s),  and  the  appropriate  mode(s)  must  be  selected.  For  example,  in 
order  to  comply  with  the  clearance  “fly  heading  180°”,  the  pilot  must  set  180  in  the  heading 
window  and  select  the  heading  mode  by  depressing  the  top  of  the  heading  selector  knob.  There  is 
a  significant  degree  of  coupling  between  the  FMC  and  MCP,  as  well  as  between  autopilot 
modes.  Some  autopilot  and  autothrottle  modes  automatically  activate  other  associated  modes, 
while  some  information  entered  into  the  FMC  will  not  be  acted  upon  unless  the  appropriate 
autopilot  mode  is  selected  on  the  MCP.  For  example,  LNAV  (lateral  navigation)  and  VNAV 
(vertical  navigation)  modes  must  be  selected  on  the  MCP  in  order  for  the  autopilot  to  follow  the 
horizontal  and  vertical  guidance  commands  entered  into  the  FMC.  Additionally,  in  some  cases 
system  behavior  depends  on  the  values  set  in  both  the  FMC  and  MCP.  For  example,  when 
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descending  in  the  VNAV  autopilot  mode,  the  controlling  altitude  will  be  the  highest  of  either  the 


altitude  set  in  the  MCP  or  an  altitude  restriction  set  in  the  FMC. 


Notes 

*  Billings,  Charles  E.  Aviation  automation:  The  search  for  a  human  centered  approach. 
Mahwah,  NJ:  Lawrence  Erlbaum  Associates,  1997. 
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